[ML] Run QA and PyTorch suites concurrently in a single triggered build by edsavage · Pull Request #3131 · elastic/ml-cpp

edsavage · 2026-07-28T22:57:58Z

Summary

When both the QA (ml_cpp_pr) and PyTorch suites are requested on a PR, trigger a single downstream appex-qa build with a comma-separated QAF_TESTS_TO_RUN instead of two separate trigger steps.

Previously the two suites were triggered as independent downstream builds. Because the appex-qa-stateful-custom-ml-cpp-build-testing pipeline has skip_queued_branch_builds=true, two builds on the same branch collided and one suite was silently skipped. Emitting a single trigger with both markers lets the appex-qa generator fan them out into concurrent parallel jobs.

Changes

.buildkite/pipeline.json.py: merge the run_qa_tests / run_pytorch_tests trigger blocks into one. QAF_TESTS_TO_RUN is built from the requested suites:
- QA only → ml_cpp_pr
- PyTorch only → pytorch_tests
- both → ml_cpp_pr,pytorch_tests

The separate PyTorch allowlist-validation step is unaffected.

⚠️ Ordering dependency

This is Part B. It depends on Part A — the appex-qa generator change in elastic/qaf-tests#309 — which teaches the generator to split a comma-separated QAF_TESTS_TO_RUN into parallel jobs. qaf-tests#309 must merge first. Until then, requesting both labels would pass ml_cpp_pr,pytorch_tests to the old generator, which treats it as a single invalid pytest -m expression. Single-suite requests are unaffected.

Testing

Local dry-runs of the generator confirm valid pipeline JSON and correct QAF_TESTS_TO_RUN for QA-only, PyTorch-only, and both-labels cases.
End-to-end validation of concurrent jobs performed on a qaf-tests feature branch build.

When both ci:run-qa-tests and ci:run-pytorch-tests are requested, emit a single Appex QA trigger passing a comma-separated QAF_TESTS_TO_RUN so the appex-qa generator fans them into concurrent parallel jobs, instead of two separate triggered builds that collided under the appex pipeline's skip_queued_branch_builds (one was skipped). Single-suite behaviour is unchanged.

elasticsearchmachine · 2026-07-28T22:58:45Z

Pinging @elastic/ml-core (Team:ML)

Copilot

Pull request overview

This PR updates the ml-cpp PR Buildkite pipeline generator to avoid QA and PyTorch downstream build collisions by triggering one appex-qa downstream build when both suites are requested, using a comma-separated QAF_TESTS_TO_RUN so the downstream generator can fan out suites into parallel jobs.

Changes:

Merge the QA and PyTorch downstream trigger logic into a single trigger step gated on run_qa_tests || run_pytorch_tests.
Build QAF_TESTS_TO_RUN as ml_cpp_pr, pytorch_tests, or ml_cpp_pr,pytorch_tests depending on requested suites.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…gger step Co-authored-by: Cursor <cursoragent@cursor.com>

edsavage · 2026-07-28T23:21:25Z

addressed both of copilot's comments in 5f74468:

Comma-separated QAF_TESTS_TO_RUN override → duplicated suites (line 96): the env override is now split on commas into individual suites, and the final list is de-duplicated while preserving order (dict.fromkeys). So both-labels + an override of ml_cpp_pr,pytorch_tests now yields ml_cpp_pr,pytorch_tests rather than ...,pytorch_tests,pytorch_tests.
Misleading step label (line 97): renamed to "Upload QA/PyTorch tests runner pipeline" since the single trigger now covers QA and/or PyTorch. (The label is display-only — generate_step doesn't derive a key from it and nothing depends on it.)

Verified with local dry-runs: QA-only, PyTorch-only, both, both + a pytorch-inclusive override (de-dups), and a QA subset override — all produce the expected QAF_TESTS_TO_RUN.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

.buildkite/pipeline.json.py:107

For PyTorch-only runs (config.run_pytorch_tests true, config.run_qa_tests false) this change switches the runner upload to run_qa_tests.yml.sh. That script emits steps labeled/commit-statused as “QA Tests”, which is misleading and also changes the commit status context compared to the previous run_pytorch_tests.yml.sh behavior. Consider keeping run_pytorch_tests.yml.sh for the PyTorch-only case and using run_qa_tests.yml.sh only when QA is involved (QA-only or QA+PyTorch).

                # De-duplicate while preserving order so an override that already
                # includes pytorch_tests does not produce a duplicate suite.
                env["QAF_TESTS_TO_RUN"] = ",".join(dict.fromkeys(qa_suites))
                pipeline_steps.append(pipeline_steps.generate_step("Upload QA/PyTorch tests runner pipeline",
                                                                   ".buildkite/pipelines/run_qa_tests.yml.sh"))

The consolidated single trigger uploaded run_qa_tests.yml.sh for every combination, so a PyTorch-only or combined run was mislabelled "QA Tests" and posted the QA commit-status context. Derive a descriptor (QA / PyTorch / QA + PyTorch) from QAF_TESTS_TO_RUN and use it for the step label, echo and github_commit_status context. Substring matching keeps it correct for marker expressions like "ml_cpp_pr and not slow". Addresses a Copilot review note on elastic#3131. Co-authored-by: Cursor <cursoragent@cursor.com>

edsavage · 2026-07-31T01:18:53Z

Addressed the low-confidence note from the latest Copilot review (re: PyTorch-only runs being labelled/commit-statused as "QA Tests").

Rather than partially reverting to run_pytorch_tests.yml.sh for the PyTorch-only case (which would reintroduce two code paths and drop the ES_BRANCH/STACK_VERSION forwarding that run_qa_tests.yml.sh provides), run_qa_tests.yml.sh now derives a descriptor from QAF_TESTS_TO_RUN and uses it for the step label, echo, and github_commit_status context:

ml_cpp_pr -> Trigger Appex QA Tests
pytorch_tests -> Trigger Appex PyTorch Tests
ml_cpp_pr,pytorch_tests -> Trigger Appex QA + PyTorch Tests

Substring matching keeps it correct for marker expressions (e.g. ml_cpp_pr and not slow). Note these contexts are not required status checks on any branch, so the rename has no branch-protection impact. Verified via dry-run across all cases; emitted YAML parses.

elastic-vault-github-plugin-prod · 2026-07-31T02:58:47Z

💔 Some backports could not be created

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts
✅	9.4
✅	9.5

Manual backport

To create the backport manually run:

backport --pr 3131

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

elastic-vault-github-plugin-prod · 2026-07-31T02:59:07Z

💔 All backports failed

Status	Branch	Result
❌	8.19	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

backport --pr 3131

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

…ld (#3131) (#3133) When both ci:run-qa-tests and ci:run-pytorch-tests are requested, emit a single Appex QA trigger passing a comma-separated QAF_TESTS_TO_RUN so the appex-qa generator fans them into concurrent parallel jobs, instead of two separate triggered builds that collided under the appex pipeline's skip_queued_branch_builds (one was skipped). Single-suite behaviour is unchanged. (cherry picked from commit 420b561) Co-authored-by: Ed Savage <ed.savage@elastic.co>

…ld (#3131) (#3132) When both ci:run-qa-tests and ci:run-pytorch-tests are requested, emit a single Appex QA trigger passing a comma-separated QAF_TESTS_TO_RUN so the appex-qa generator fans them into concurrent parallel jobs, instead of two separate triggered builds that collided under the appex pipeline's skip_queued_branch_builds (one was skipped). Single-suite behaviour is unchanged. (cherry picked from commit 420b561) Co-authored-by: Ed Savage <ed.savage@elastic.co>

…ed build (#3131) (#3134) Manual 8.19 backport of #3131. The auto-backport conflicts because 8.19 lacks the "Upload ES inference tests x86_64" step that sits adjacent to the QA-trigger block on main, so the hunk context cannot anchor. Consolidates the separate run_qa_tests / run_pytorch_tests upload steps into a single trigger with a comma-separated QAF_TESTS_TO_RUN, so the appex-qa generator fans QA (ml_cpp_pr) and PyTorch suites out into concurrent parallel jobs in one downstream build (avoiding the skip_queued_branch_builds de-dup that dropped one of two separate triggers). run_qa_tests.yml.sh now derives its step label and commit status context (QA / PyTorch / QA + PyTorch) from the requested suites. Adds "import os" to pipeline.json.py: 8.19 did not previously import it and the consolidated block reads QAF_TESTS_TO_RUN via os.environ. Co-authored-by: Cursor <cursoragent@cursor.com>

elasticsearchmachine added the v9.6.0 label Jul 28, 2026

edsavage added :ml >build labels Jul 28, 2026

edsavage requested a review from Copilot July 28, 2026 23:05

Copilot started reviewing on behalf of edsavage July 28, 2026 23:06 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

Comment thread .buildkite/pipeline.json.py Outdated

Comment thread .buildkite/pipeline.json.py Outdated

Address Copilot review: de-dup QAF_TESTS_TO_RUN suites and rename tri…

5f74468

…gger step Co-authored-by: Cursor <cursoragent@cursor.com>

edsavage requested a review from Copilot July 28, 2026 23:37

Copilot started reviewing on behalf of edsavage July 28, 2026 23:37 View session

edsavage requested a review from wwang500 July 28, 2026 23:38

Copilot AI reviewed Jul 28, 2026

View reviewed changes

edsavage added v8.19.20 v9.4.5 v9.5.1 auto-backport Automatically merge backport PRs when CI passes labels Jul 28, 2026

edsavage mentioned this pull request Jul 29, 2026

[ML] Fix backport-pending label handling for auto-backports #3128

Merged

5 tasks

edsavage merged commit 420b561 into elastic:main Jul 31, 2026
24 checks passed

elastic-vault-github-plugin-prod Bot mentioned this pull request Jul 31, 2026

[9.4] [ML] Run QA and PyTorch suites concurrently in a single triggered build (#3131) #3132

Merged

elastic-vault-github-plugin-prod Bot added the backport-pending label Jul 31, 2026

elastic-vault-github-plugin-prod Bot mentioned this pull request Jul 31, 2026

[9.5] [ML] Run QA and PyTorch suites concurrently in a single triggered build (#3131) #3133

Merged

edsavage mentioned this pull request Jul 31, 2026

[8.19] [ML] Run QA and PyTorch suites concurrently in a single triggered build (#3131) #3134

Merged

github-actions Bot removed the backport-pending label Aug 2, 2026

Navigation Menu

Uh oh!

[ML] Run QA and PyTorch suites concurrently in a single triggered build - #3131

[ML] Run QA and PyTorch suites concurrently in a single triggered build#3131
edsavage merged 3 commits into
elastic:mainfrom
edsavage:ci/qa-pytorch-concurrent-jobs

edsavage commented Jul 28, 2026 •

edited

Loading

Uh oh!

elasticsearchmachine commented Jul 28, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

edsavage commented Jul 28, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

edsavage commented Jul 31, 2026

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 31, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

edsavage commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

⚠️ Ordering dependency

Testing

Uh oh!

elasticsearchmachine commented Jul 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

edsavage commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

edsavage commented Jul 31, 2026

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 31, 2026

💔 Some backports could not be created

Manual backport

Questions ?

Uh oh!

elastic-vault-github-plugin-prod Bot commented Jul 31, 2026

💔 All backports failed

Manual backport

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

edsavage commented Jul 28, 2026 •

edited

Loading

edsavage commented Jul 28, 2026 •

edited

Loading